feat(lmeval): Disable online evaluation #375
Conversation
feat(lmeval): Add offline mode as mandatory
ruivieira
requested review from
danielezonca,
yhwang,
RobGeada,
resoluteCoder,
christinaexyou and
m-misiura
|
PR image build and manifest generation completed successfully! 📦 PR image: 📦 LMES driver image: 📦 LMES job image: 🗂️ CI manifests |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: RobGeada The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@ruivieira: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
* feat(lmeval): Add offline mode as default (#370) Refer to [RHOAIENG-16200](https://issues.redhat.com/browse/RHOAIENG-16200) * feat(lmeval): Disable remote code execution (#371) Refer to [RHOAIENG-16203](https://issues.redhat.com/browse/RHOAIENG-16203) * feat(lmeval): Disable online evaluation (#373) * feat(lmeval): Add offline mode as mandatory * feat(lmeval): Add online and code execution flags * Update images * Add default to flags * Restore images (cherry picked from commit bf751f5)
Refer to RHOAIENG-16382.
This PR adds two flags to the LMEval job:
allowOnline: true/false, defaults to false if missing. If false, lm-eval and HF libraries will not automatically download models or datasets if they are missing locally. If explicitly set to true, models and datasets will be downloaded.allowCodeExecution: true/false, defaults to false if missing. If false, scripts present in the models, datasets or metrics will not be executed. If true, they will.